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Abstract 



This session will describe a process used to assess the technical 
competency of non-direct line candidates for managerial positions in 
computer specialties. A multi-method approach, combining task-based, 
computer scorable evaluations of training and experience with multiple- 
choice batteries, was used as a qualifying screen for determining 
candidates’ knowledge, training, and experience in specific competency 
areas. An assessment matrix was used to evaluate candidate scores 
from the multi-methods and identify candidates in need of follow-up. 
The process was content valid, efficient (N of candidates across all 
competencies equals 1200) and acceptable to candidates and subject 
matter experts. 



Background 

There are six titles involved in this promotion series to second level supervisory positions 
where the principal responsibility is the management of a technical EDP operation. The generic title 
is Manager of Data Processing Services. All individuals in first-line supervisory computer positions 
qualified for the generic title. The series also included one generic/technical title and four technical 
parenthetics. Direct-line candidates in lower level technical titles are "A-list" eligibles for the 
corresponding technical Manager titles; a qualifying screen was developed to determine "B-list" 
eligibility for these technical specialties.’ Candidates who qualified for any of the technical 
parenthetic titles also qualified for the generic/technical title. Table One illustrates the organizational 
relationship between a sample of these titles. The computer administered simulation test was the sole 
ranking portion for all titles in this series. 

Examinations for these titles were last held in 1984-85. That selection plan consisted of an 
oral only, and only direct-line promotions were allowed. Although the need to hold examinations 
for these titles had been apparent for some time, because of financial and staff constraints, the 
Department lacked the resources to conduct oral examinations for this large of a candid-^tf 
competition. After considerable negotiation, the Testing Services Division agreed to develop a 
simulation examination, which was to be administered via computer. 

Extensive consultations with the Interdepartmental Committee on Electronic Data Processing 
(ICEDP) and the New York State Forum on Information Resource Management (NYSFIRM) 
revealed that these organizations wanted to increase the ability of candidates to qualify tor more than 
one title. For example, they felt that there may be some individuals who, "... are in an Associate 
Data Communications title who could perform the duties of the Manager of Data Base.' 
Concomitantly, they insisted that B-list eligibles need to have demonstrated competency in the 
technical specialty. 



' Simply put, A-List eligibles are those individuals who qualify for the examination because they 
serve in the same title at the next lower level. For example, an Assoeiate Computer Programmer 
(Data Base) in an A-List eligible for the Manager of Data Processing (Data Base). Hc/she would 
be a B-List eligible for Manager of Data Processing (Data Communications) positions. "A-list 
eligibles have preference over B-list eligibles. 



The Assessment Matrix 



In response to this mandate, the Testing Service Division developed the "matrix" approach 
to evaluating competency in each technical specialty. The premise behind this model was to provide 
a qualifying screen which would allow candidates to demonstrate their competency in each technical 
area utili2dng multiple assessment devices. The matrix consisted of two principal components--a 
multiple-choice battery and a rating of the candidates’ training and experience. Each component 
focused directly on the technical area. Table Two provides a generic example of the Assessment 
Matrix. The testing measure used is displayed in the rows, the competency score in the columns, 
and standards established for obtaining that score in the cells. As can be seen, each cell is scored 
as a 0, 1 or 2 depending upon the competency level demonstrated. The sum of the cell scores (this 
is a full compensatory model) is the candidate's competency score.' For each title, the multiple- 
choice components comprised half of the weight of the matrix; the assessment of training and 
experience was equally weighted with the multiple-choice assessments. In general, a score of four 
represents "minimal competence demonstrated." 

The Multiple-Choice Component - Standards and Scoring 

The multiple-choice contributions to the matrix are straightforward. A test battery for the 
Associate level generic and technical titles was administered in April 1993. The test for each title 
included two multiple-choice subtests of 15 items each assessing applied knowledge in each technical 
specialty. To be considered as B-Iist eligibles for a Manager title candidates were required to take 
the specialty-specific multiple-choice material. 

After consultation with ICEDP and NYSFIRM it was decided that below average scores (less 
than the mean) were not an adequate demonstration of minimal competence and would be assigned 
a matrix cell score of 0 points. Scores at, or higher than the mean, but less than one standard 
deviation above the mean, equated with minimal competence and would be assigned a matrix cell 
score of one point. Scores one standard deviation above the mean, or higher, are indicative of 
exceptional competence in the specialty area and would be assigned a matrix cell score of 2 points. 
These standards were uniformly applied for each subtest in each specialty area for each title. Tables 
Three A and Three B show frequency distribution data for each of the subtests. Tables Four A and 
Four B provide assessment matrix score data for candidate performance on the multiple-choice 
components.^ NOTE: Fifty Seven candidates completed the E&T Questionnaire but did not take the 
corresponding multiple choice subtest. The multiple choice tables record these candidates as missing. 
They are included in the data analysis for the E&T component. 

Training and Experience Component - Design and Scoring 

The assessment of the Experience and Training (E&T) portion ol the candidates 
competencies was more complex. The operational objective was to develop a machine readable 
(opscannable) questionnaire (N of ratings = 1200) which would provide the data upon which E&T 
competency decisions would be based. Using the job analysis information and working with 



’Conceptually, the Assessment Matrix could be extended along each vector. Additional assessment 
devices could be added; less aggregate values than 0. 1 or 2 could be assigned; each cell could be 
weighted in proportion to the criticality of .successful performance in the job. 

' fables Three through Ten used the data obtained from the Manager of Data Processing (Data Base) 
as an example. Similar re.sults were obtained and decisions were made for each of the titles 
evaluated. 
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committees of subject matter experts in each specialty area. Testing Division staff determined those 
20 critical tasks which uniquely defined competence in the technical specialty. For example, 
although "writes computer programs in COBAL" might be a critical task for the data base title, this 
task is not unique to successful performance in the data base specialty. On the other hand, "Defines 
data elements for new data base systems and/or normalizes data elements" is a critical task done only 
in data base environments. Experience in the latter task would be rated; the former task would be 
excluded from the analysis. 

Next, a questionnaire was prepared which asked for information about the candidates' 
experience in performing each task in four dimensions. First, has the respondent performed the task 
since April of 1991?* Second, how often had the respondent performed this task (indicated on a 
frequency scale of 1 through 15+)? Third, how competent is the respondent in performing the task 
(provided by a self-rated competence scale ranging from not familiar to master)? Finally, candidates 
were required to provide the names of individuals who could verify the respondent s answers to the 
first three questions. A sample of the questionnaire and the opscannaole form are attached. 

The combination of the data and the derivation of the matrix cell scores for the experience 
portion was straightforward. A task competency score (0 or 1) was derived for each candidate lor 
each task. 

Candidates were assigned a task competency score of 0 if; 

(1) They had not performed the task since April 17, 1991; or 

(2) They performed the task, but did so less than three times' (frequency equals 0. 1 or 
2); or, 

(3) They indicated that they were either not familiar with the task or would need guidance 
in performing the task (competency level equals 1 or 2). 

Candidates were assigned a task score of 1 if: 

(1) They performed the task since April 1991; and, 

(2) They performed the task 3 or more times (frequency > 3); and. 

(3) They indicated that they could perform the task independently or were a master at 
performing the task (competency level equals 3 or 4); and. 

(4) They provided the names of individuals who could verify their frequency and 
competence in the performance of the task. 

Table Five A. B, and C provide data which illustrate candidate responses to task Four. Table 
Six provides a crosstabulation of the frequency by competency scores. Table Seven shows the 
summary results of this data reduction. 



■’ The April 1991 date was selected by the ICEDP and subject matter expert committees as the cutoff 
date for relevant experience. Their evaluation was that, because of the rapidly changing technology 
involved with each of these specialties, the quality of experience greater than 2 1/2 years old may 
very likely be outmoded. 



"This standard was based on the premise that performance of the task two or fewer times since April 
1991 meant it was performed, on average, less than once a year. 



The sum of these task competency scores yielded, for each candidate, the number of tasks 
he/she could competently perfonu— the component competency score. This component competency 
score ranged from 0 (competent in no tasks) to 20 (competent in every task). 

In consultation with ICEDP, NYSFIRM and subject matter experts, it was decided that any 
candidate who was competent in fewer than half of the tasks (component competency score < 9) did 
not demonstrate minimal competence in the technical specialty and would be assigned a matrix cell 
score of 0. Candidates who could competently perform one half or more but fewer than three 
quarters of the tasks (component competency score > 10 < 15) were assigned the matrix cell score 
of 1. Any candidate who was competent in three quarters or more of the tasks (> 15) was assigned 
a matrix cell score of 2. These standards were applied to the experience ratings of all candidates for 

all titles. 

Table Eight displays the frequency distribution of the c'^mponent competency scores and the 
corresponding matrix score. 

The training component of the matrix was developed using a more traditional approach. 
Table Two illustrates how the points were assigned to candidates in this portion of the matrix. 
Candidates achieved scores of 0, 1 or 2 based upon their combinations of relevant experience. These 
standards were developed in consultation with the SME groups. As with the other components in 
the matrix, with minor differences, the same standards were applied across all four of the technical 
titles of interest. 

Tables Nine A and B provide a summary of the experience and training matrix score by 
component. As can be seen, each component contributed to assessing the competency of interest. 
In the general model, a matrix score of four or higher was the standard set for qualifying tor the 
technical examination. The basis for this standard was that if a candidate demonstrated minimal 
competence in each of the four assessment devices he or she would have a matrix score of four. 
However, as indicated earlier, the matrix is a full compensatory model-a less than minimal score 
on one dimension could be offset by an exceptional score in another. 

Tables Ten A and Ten B provide summary matrix score data for the Data Base Manager 
candidates. 



Differences By Examination 

The assessment matrices for the Manager of Data Processing and the Manager oi Data 
Communications were both straightforw'ard in content and design. The qualifying matrix for each 
of these qualifying examinations contained two- 15 item multiple-choice components, and the ratings 
experience and training. The obtainable scores ranged from 0 to 8; a score of four or higher was 
considered qualifying. The matrix for the Manager System Programmer title contained three 
multiple-choice subtests in addition to the E&T ratings; therefore, in order to equally weight the 
multiple-choice versus E&T component, each multiple-choice matrix score was multiplied by two 
and each E&T score by three to yield a total possible matrix score of 24; the standard for minimum 
competence set at 1 2. Similarly, the Operations Manager examination contained four multiple-choice 
subte.sts. To equate the E&T and multiple-choice portions we multiplied each E&f cell score by 
two; therefore, the obtainable scores ranged from 0 to 16 and the minimum qualifying score was set 
at 8. 



Tables Eleven, Twelve and Thirteen (A and 13) provide summary data comparable to the 
Manager of the Data Base title. 



Results and Standards 



Tables Ten A through Thirteen A display the results of the Assessment Matrix by the number 
of candidates within each cell. There is a high degree of consistency in the proportion of candidates 
within corresponding cells. A marginally greater proportion of candidates performed better on the 
multiple-choice than the E&T components. Given the known reliability of the multiple-choice tests 
this is satisfactory. 

A review of Tables Ten B through Thirteen B shows that the percentage of B-list eligibles 
passing this qualifying screen ranges from 20% to 26% of the eligibles. Further, approximately 40 
to 50% of these individuals will not pass the simulation examination; hence, the number of eligibles 
on the B-lists is anticipated to be very small. As indicated earlier, the subject matter experts 
representing the ICEDP and the NYSFIRM were unequivocal in the position that it was critical that 
any B-list eligible be competent in the technical specialty. Therefore, it was essential that standards 
be kept high. Also, the selection ratio for these positions is extremely low; recruitment estimates 
indicate that, over the course of the next four years, only 4 or 5 appointments wall be made to each 
of the four technical parenthetic positions. ICEDP predicts that the number of appointments made 
of B-list eligibles will be few or none. 

Verification of the Candidate Responses 

Table Fourteen displays the cross-tabulations of matrix scores by component. Inconsistency 
in candidate responses targeted selected individuals for further verification. First, any candidate with 
a matrix score less than 4 w'as excluded from further investigation. Candidates who achieved high 
multiple choice scores (X>3), and correspondingly high E&T scores, (X>2) w^ere also excluded trom 
further review. Verifiers were contacted for each candidate w'ho qualified primarily based upon their 

score on the E&T (N=30). 



Psychometric Analysis 

The testing components used in assessing the candidates’ technical qualifications tor B-list 
eligibility formed the basis for their eligibility to be scored on the ranking portion of the 
examination. All candidates qualified as "A-list" eligibles for at least one Data Processing Manager 
title. A job simulation examination consisting of two interactive problems was administered-via 
computer— to approximately eleven hundred candidates. The first problem centered on the 
supervisory and management skills necessary for successful performance in managing an EDP 
operation; the second problem centered on the candidates’ ability to manage a complex EDP project. 
Each problem focused on the candidates' problem-solving, decision-making, information-gathering 
and interpersonal skills. 

Although the multiple-choice, experience and training distributions were evaluated 
dichotomously for the purposes of the matrix, the underlying distributions of the first two 
components, and to a lesser extent, the training component, were continuous and could, conceivably, 
he used as ranking mechanisms, l ables Fifteen A and B provide task competency score descriptive 
data (means, standard deviations, number of cases and reliability estimates) for the experience 
component. Since Table Fifteen A includes those candidates who did not perform a particular task, 
the reliahility estimate may be spuriously high. Table Fifteen B excludes those candidates from the 
analysis. An examination of the data shows considerable variability in the task score item means 
and high reliability in the scoring mechanism. Indeed, although difficult, the di.strihution statistics 
are very similar to those obtained using multiple-choice tests. 



Table Sixteen presents the intercorrelation matrix among the variables in the assessment 
system. Most of the correlations are zero order; however, the significant correlations are as expected. 
The multiple-choice subtests are highly inter-correlated. Since the number of training courses 
(Crednum) and number of training hours reflect clustered rather than continuous distributions, their 
correlations with the other components are predictably low. The simulation problems (Sim 1 and 
Sim 2) correlate higher with each other than with the other variables-again, as anticipated, since they 
were designed to evaluate managerial, rather than technical, skills. 

Table Seventeen presents a Principal Components Factor Analysis with Varima.\ rotation for 
the seven variables. A two-factor solution, accounting for forty-four percent of the variance, was 
derived. The loading of the components on these two factors can reasonably be described as 
indicative of a technical dimension and a managerial dimension. 
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TABLE TWO 
GENERIC MATRIX 





CO^ 

0 


4PETENCY DEMONSTRATE! 
1 


D 

2 


MCI 


Below Associate Mean Score 


Mean to One Standard Deviation 


One Standard Deviation or 
More Above Mean 


MC2 


Below Associate Mean Score 


Mean to One Standard Deviation 


One Standard Deviation or 
More Above Mean 


liXPlilllliNCL 


Competent in E'ewer Ilian 
I'en Easks 


Competent in More Than 1/2 But 
Less Than 3/4 of Tasks 


Competent in 3/4 or More 
of Tasks 


TRAINING 


Anytiiing ixss I'lian Shown 
in 1 


6 College Credits 

or 

24 Hours Training 

or 

3 Credits + 12 Hours Training 


6 Credits + 24 hours 
or 

48 Hours 

or 

3 Credits 36 Hours 

or 

12 Credits 



O 

ERIC 



TABLE THREE A 

MULTIPLE CHOICE SUBTEST ONE 
DATA BASE MANAGER 



Score 


Subtest Score 


Frequency 


Percent 




3 


2 


.5 




4 


10 


2.6 


0 


5 


21 


5.5 




6 


33 


8.7 




7 


43 


11.3 




8 


57 


15.0 


1 


9 


48 


12.7 




10 


52 


13.7 




11 


33 


8.7 




12 


14 


3.7 


2 


13 


6 


1.6 




14 


n 


.5 




15 


1 


.3 






57 


15.0 




TOTAL 


379 


100.0 



Mean = 8.416 
Std Dev = 2.209 




TABLE THREE B 

MULTIPLE CHOICE SUBTEST TWO 
DATA BASE MANAGER 



Matrix Score 


Subtest Score 


Frequency 


Percent 




1 


4 


1.1 




3 


9 


2.4 




4 


33 


8.7 


0 


5 


32 


8.4 




6 


50 


13.2 




7 


57 


15.0 


1 


8 


45 


11.9 




9 


34 


9.0 




10 


27 


7.1 




11 


17 


4.5 




12 


8 


2.1 


2 


13 


n 


.8 




14 


.3 


.8 






57 


15.0 




TOTAL 


379 


100.0 



Mean = 7.224 
Std Dev - 2.403 




TABLE FOUR A 
MATRIX SCORE 

MULTIPLE CHOICE SUBTEST ONE 
DATA BASE MANAGER 



Matrix Score 


Frequency 


Percent 


0 


166 


43.8 


1 


100 


26.4 


2 


56 


14.8 




57 


15.0 


TOTAL 


379 


100.0 



Mean = .658 
Std Dev = .758 



TABLE FOUR B 
MATRIX SCORE 

MULTIPLE CHOICE SUBTEST TWO 
DATA BASE MANAGER 



Matrix Score 


Frequency 


Percent 


0 


185 


48.8 


1 


79 


20.9 


2 


58 


15.3 




57 


15.0 


TOTAL 


379 


100.0 



Mean = .606 
Std Dev = .775 



TABLE FIVE A 

DO/DO NOT PERFORM TASK 04 
DATA BASE MANAGER 





Value 


Frequency 


Percent 


DO NOT PERFORM 


0 


179 


47.2 


DO PERFORM 


1 


200 


52.8 




TOTAL 


379 


100.0 



TABLE FIVE B 

FREQUENCY OF PERFORMANCE TASK 04 
DATA BASE MANAGER 



Frequency 


No. of 
Respondents 


Percent 


0 


179 


47.2 


1 


18 


4.7 


2 


34 


9.0 


3 


26 


6.9 


4 


16 


4.2 


5 


25 


6.6 


6 


10 


2.6 


7 


2 


.5 


8 


4 


1.1 


10 


15 


4.0 


12 


3 


.8 


13 




.5 


15 


45 


11.9 





TABLE FIVE C 

COMPETENCY LEVEL - TASK 04 
DATA BASE MANAGER 



Competency 


Value 


No. of 
Respondants 


Percent 


NOT PERFORM 


0 


179 


47.2 


NOT FAMILIAR 


1 


3 


.8 


NEED GUIDANCE 


2 


32 


8.4 


INDEPENDENTLY 


3 


122 


32.2 


MASTER 


4 


43 


11. 3 




TOTAL 


379 


100.0 
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TABLE SIX 

FREQUENCY OF PERFORMANCE BY 
COMPETENCY - TASK 04 
DATA BASE MANAGERS 



FREQUENCY 


NOT 

PERFORM 


NOT FAM 


NEED 

GUIDE 


INDEPENDENT 


MASTER 


TOTAL 


0 


178 






1 




179 


1 




1 


7 


9 


1 


18 


2 




1 


9 


21 


3 


34 


3 




1 


6 


17 


2 


26 


4 






3 


10 


3 


16 


5 


1 




3 


17 


4 


25 


6 






2 


5 


3 


10 


7 








2 




2 


8 








3 


1 


4 


10 






1 


14 




15 


12 






1 


1 


1 


3 


13 








1 


I 


2 


15 








21 


24 


45 


TOTAL 


179 


3 


32 


122 


43 


379 


% 


47.2 


8 


8.4 


32.2 


11.3 


100.0 



TABLE SEVEN 

TASK 04 COMPETENCY SCORE 
DATA BASE MANAGER 



Task Score 


Frequency 


Percent 


0 


249 


65.7 


1 


130 


34.3 


TOTAL 


379 


100.0 



Mean = .343 
Std Dev = .475 
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TABLE EIGHT 

TASK COMPETENCY SCORES 
FREQUENCY AND COMPETENCE ACROSS 20 TASKS 
DATA BASE MANAGER 



Matrix Score 


Task 

Competency 

Score 


Frequency 


Percent 


0 


0 


153 


40.4 




1 


23 


6.1 




2 


22 


5.8 




3 


17 


4.5 




4 


15 


4.0 




5 


12 


3.2 




6 


11 


2.9 




7 


16 


4.2 




8 


15 


4.0 




9 


8 


2.1 


1 


10 


10 


2.6 




11 


8 


2.1 




12 


9 


2.4 




13 


9 


2.4 




14 


8 


2.1 


2 


15 


6 


1.6 




16 


9 


2.4 




17 


5 


1.3 




18 


4 


1.1 




19 


8 


2.1 




20 


11 


2.9 




lOTAL 


379 


100.0 





TABLE NINE A 
EXPERIENCE 
DATA BASE MANAGER 



Matrix Score 


Frequency 


Percent 


0 


292 


77.0 


1 


44 


11.6 


2 


43 


11.3 


TOTAL 


379 


100.0 



Mean = .343 
Std Dev = .673 



TABLE NINE B 
TRAINING 

DATA BASE MANAGER 



Matrix Score 


Frequency 


Percent 


0 


260 


68.6 


1 


66 


17.4 




53 


14.0 


TOTAL 


379 


100.0 



Mean = .454 
Std Dev = .727 











TABLE TEN A 

NUMBER OF CANDIDATES IN EACH MATRIX CELL 
DATA BASE MANAGER 





COMPETEN 

0 


CY DEMONS' 
1 


PRATED 

2 


MCI 


166 


100 


56 


MC2 


185 


79 


58 


EXPERIENCE 


292 


44 


43 


TRAINING 


260 


66 


53 



TABLE TEN B 

DISTRIBUTION OF MATRIX SCORES 
DATA BASE MANAGER 



SCORE 


FREQUENCY 


CUM FREQ 


% 


8 


I 


1 




7 


5 


6 


1.6 


6 


13 


19 


4.0 


5 


16 


35 


5.0 


4* 


28 


63 


8.7 


3 


52 


115 


16.1 


2 


61 


176 


18.9 


1 


64 


240 


19.9 


0 


82 


322 


25.5 



Note: Excludes Candidates Who Did Not Participate in All Components 
’''Qiiulifying Score 
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TABLE ELEVEN A 

NUMBER OF CANDIDATES IN EACH MATRIX CELL 
DATA COMMUNICATIONS MANAGER 





COMPETENCY 


DEMONSTRATED 




0 




1 


2 


MCI 


119 


112 


49 


MC2 


130 


99 


51 


EXPERIENCE 


273 


29 


34 


TRAINING 


279 


28 


29 



TABLE ELEVEN B 

DISTRIBUTION OF MATRIX SCORES 
DATA COMMUNICATIONS MANAGER 



SCORE 


FREQUENCY 


CUM FREQ 


% 1 


8 


1 


1 


.4 


7 


3 


4 


1.1 


6 


14 


18 


5.0 


5 


18 


36 


6.4 


4 * 


26 


62 


9.3 


3 


25 


87 


8.9 


2 


51 


138 


18.2 


1 


61 


199 


21.8 


0 


81 


280 


28.9 



Note: Excludes Candidates Who Did Not Participate in All Components 
^Qualifying Score 
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TABLE TWELVE A 

NUMBER OF CANDIDATES IN EACH MATRIX CELL 
OPERATIONS MANAGER 





COMPETEN 

0 


CY DEMONS' 
1 


PRATED 

2 


*MC1 


93 


123 


62 


*MC2 


152 


93 


J J 


*MC3 


117 


141 


20 


*MC4 


103 


1 14 


61 


"♦EXPERIENCE 


281 


35 


22 


""TRAINING 


259 


52 


27 



* Weighted 1 
“Weighted 2 



TABLE TWELVE B 
DISTRIBUTION OF MATRIX SCORES 
OPERATIONS MANAGER 



SCORE 


FREQUENCY 


CUM FREQ 


% 


12 


3 


3 


l.l 


11 


1 


4 


.4 


10 


8 


12 


2.9 


9 


0 


12 


0.0 


8* 


15 


27 


5.4 


7 


16 


43 


5.8 


6 


30 


73 


10.8 


5 


35 


108 


12.6 


4 


34 


142 


1 2.2 


3 


44 


186 


15.8 


2 


49 


235 


17.6 


1 


'S •> 


268 


1 1.9 


0 


10 


278 


3.6 



Note: Excludes Candidates Who Did Not Participate in All Components 
’Apualifying Score 





TABLE THIRTEEN A 

NUMBEI OF CANDIDATES IN EACH MATRIX CELL 
SYSTEMS MANAGER 





COMPETEN 

0 


CY DEMONS' 

1 


TRATED 

2 


*MC1 


119 


84 


41 


*MC2 


130 


71 


43 


*MC3 


102 


87 


51 


♦^EXPERIENCE 


212 


43 


50 


♦♦TRAINING 


218 


43 


44 



* Weighted 2 
**Weighted 3 



TABLE THIRTEEN B 
DISTRIBUTION OF MATRIX SCORES 
SYSTEMS MANAGER 



SCORE 


FREQ 


CUM FREQ 


% 


SCORE 


FREQ 


FREQ 


CUM % 


24 


1 


Q.O 


0 


11 


9 


56 


3 8 


23 


0 


0.0 


0 


10 


8 


64 


3.3 


22 




.8 


1 


9 


6 


70 


2.5 


21 


1 


0.0 


2 


8 


22 


92 


9.2 


20 


3 


1.3 


3 


7 


7 


99 


2.9 


19 


1 


0.0 


3 


6 


18 


117 


7.5 


18 


7 


2.9 


6 


5 


15 


132 


6.3 


17 


6 


2.5 


9 


4 


26 


158 


10.8 


16 


D 


.8 


10 


3 


4 


162 


1.7 


15 


3 


1.3 


11 


2 


49 


211 


20.4 


14 


1 1 


4.6 


15 


1 


0 


21 1 


0.0 


13 


6 


2.5 


18 


0 


29 


240 


12.1 


12^ 


4 


^7 


20 











111” Score 




i P i 

^ X 




TABLE FOURTEEN 
MATRIX SCORES 

TRAINING & EXPERIENCE (ROWS) 

BY MULTIPLE CHOICE TESTS (COLUMNS) 



Matrix 


0 


1 


2 




4 


Total 


0 


82 


47 


35 


" -1 


6 


187 


1 


17 


14 


15 


9 


3 


58 


2 


12 


16 ^ 


10 


9 


6 


53 


3 


4 1 


2 


1 


3 


2 


12 


4 


1 


3 


4 


3 


1 


12 


TOTAL 


116 


82 


65 


41 


18 


322 


% 


36.0 


25.5 


20.2 


12.7 


5.6 


100.0 



* Excluded from Further Verification 



Number of Missing Observations = 57 



TABLE FIFTEEN A 
RELIABILITY ANALYSIS 

TASK COMPETENCY SCORES (INCLUDING ZEROS) 







Mean 


Std Dev 


Cases 


1. 


SCORED 1 


.2296 


.4211 


379.0 


2. 


SCORE02 


.1741 


.3797 


379.0 


3. 


SCORE03 


.2691 


.4441 


379.0 


4. 


SCORE04 


.3430 


.4753 


379.0 


5. 


SCORE05 


.3694 


.4833 


379.0 


6. 


SCORE06 


.2639 


.4413 


379.0 


7. 


SCORE07 


.2375 


.4261 


379.0 


8. 


SCORE08 


.3061 


.4615 


379.0 


9. 


SCORE09 


.2559 


.4370 


379.0 


10. 


SCORE 10 


.3430 


.4753 


379.0 


11. 


SCOREll 


.2058 


.4048 


379.0 


12. 


SCORE12 


.2243 


.4177 


379.0 


13. 


SCORE13 


.2216 


.4159 


379.0 


14, 


SCORE14 


.3219 


.4678 


379.0 


15. 


SCORE 15 


.2507 


.4340 


379.0 


16. 


SCORE16 


.1953 


.3969 


379.0 


17. 


SCORE17 


.2559 


.4370 


379.0 


18. 


SCORE18 


.1214 


.3270 


379.0 


19. 


SCORE19 


.1873 


.3907 


379.0 


20. 


SCORE20 


.2269 


.4194 


379.0 



Alpha - .9489 



Standardized item alpha ~ .9490 





TABLE FIFTEEN B 
RELIABILITY ANALYSIS 

TASK COMPETENCY SCORES (EXCLUDING ZEROS) 



Mean Std Dev Cases 



1. 


SCOREOl 


.3850 


.4877 


226.0 


2. 


SCORE02 


.2920 


.4557 


226.0 


3. 


SCORE03 


.4513 


.4987 


226.0 


4. 


SCORE04 


.5752 


.4954 


226.0 


.3. 


SCORE05 


.6195 


.4866 


226.0 


6. 


SCORE06 


.4425 


.4978 


226.0 


7. 


SCORE07 


.3982 


.4906 


226.0 


8. 


SCORE08 


.5133 


.5009 


226.0 


9. 


SCORE09 


.4292 


.4961 


226.0 


10. 


SCORE 10 


.5752 


.4954 


226.0 


11. 


SCOREll 


.3451 


.4765 


226,0 


12. 


SCORE12 


.3761 


.4855 


226.0 


13. 


SCORE13 


.3717 


.4843 


226.0 


14. 


SCORE14 


.5398 


.4995 


226.0 


15. 


SCORE15 


.4204 


.4947 


226.0 


16. 


SCORE16 


.3274 


.4703 


226.0 


17. 


SCORE17 


.4292 


.4961 


226.0 


18. 


SCORE18 


.2035 


.4035 


226.0 


19. 


SCORE19 


.3142 


.4652 


226.0 


20. 


SCORE20 


.3805 


.4866 


226.0 



Alpha = .9090 Standardized item alpha = .9092 




TABLE SIXTEEN 

CORRELATION MARTIN OF COMPONENTS IN THE SELECTION SYSTEM 



MCONE MCTWO CREDNUM TRAINHRS TASKSCOR SIMl 



MCONE 


1.00000 






MCTWO 


.40821 


1.00000 




CREDNUM 


.08186 


.25925 


1.00000 


TR.AINHRS 


.15483 


.13039 


.05004 


TASKSCOR 


.18907 


.16953 


.15570 


SlMl 


.15547 


.15942 


.09845 


S1M2 


.05193 


.09356 


-.00769 



1.00000 

.24143 1.00000 

.02407 .01880 1.00000 

-.05411 -.00142 .22695 



TABLE SEVENTEEN 

FACTOR ANALYSIS OF VARIABLES IN THE SELECTION SYSTEM 



Initial Statistics: 



Vai’iable 


Communality 


* 


Factor 


Eigenvalue 


Pet of Var 


Cum Pet 


S416H0 


1.00000 




1 


1.84026 


26.3 


26.3 


S416H1 


1.00000 




2 


1.24323 


17.8 


44.0 


CREDNUM 


1.00000 


♦ 


3 


.97485 


13.9 


58.0 


TRAINHRS 


1.00000 




4 


.89928 


12.8 


70.8 


TASKSCOR 


. 1.00000 


* 


5 


.78947 


11.3 


82.1 


SIMl 


1.00000 




6 


.71315 


10.2 


92.3 


SIM2 


1.00000 


* 


7 


.53976 


7.7 


100.0 



PC extracted 2 factors. 



VARIMAX Rotated Factor Matrix: 





Factor 1 


Factor 2 


S416H0 


.63396 


.24281 


S416H1 


.66603 


.32001 


CREDNUM 


.45038 


.10005 


TRAINHRS 


.55005 


-.30076 


TASKSCOR 


.62296 


-.20778 


SIMl 


.16480 


.69818 


SIM2 


-.05370 


.72249 
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EXAM NO. (Mark only ONE.) TITLE 



i 32-709 Manager of Data Processing Technical 
Services (Data Base) 



' ' 32-71 0 Manager of Data Processing Technical 
' Services (Data Communications) 



32-711 Maj^ager of Data Processing Technical 
Servis'^s (Operations) 



32-71 2 Manager of Data Processing Technical 
Services (Systems Processing) 



ANSWER ONLY IF YOU ANSWERED "1“ TO THE QUESTION 
REGARDING WHETHER OR NOT YOU HAVE PERFORMED THIS 







Have you 
performed 
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COMPETENCE 


VERIRER(S) 




Task 


this task 










f=^:jPOli'^NCY 






What IS your level 


Who can venfy your 


wmm 


since Apnl 
























of competence in 


level of competence 
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No. 


17, 1991’ 


How often have you performed this task since Apnl 1 7, 1 991 '? 


this task'? 
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21 Have you completed college 
credits in tnis speaality'? 

■7 = NO ■ YES 

22 If yes, how many college credits'? 

. M - 2 College Credits 
: 3 - 5 College Credits 
■ 6-11 College Credits 
: 12 or more College Credits 

23 Have you completed non-credit trainina in 
this spe' lality since Apnl 1 7, 1 991 ? 

.'j NO ^ YES 

24 If yes, how many contact hours'? 

: 1 - 11 Contact Hours 
12-23 Contact Hours 
24 - 35 Contact Hours 
. 36 - 47 Contact Hours 
; 48 or more Contact Hours 



o 



November, 1993 



Candidate Name: 



Candidate identification 
Number: 



Exam No. 32-709 

Manager of Data Processing Technical Services 

(Data Base) 



EXPERIENCE 

AND 

TRAINING 

QUESTIONNAIRE 



DIRECTIONS FOR 

EXPERIENCE PORTION OF THE QUESTIONNAIRE 

A. On pagss 4 and 5 is a form antitlod "Task Listing . It lists tasks 
that a person in the Data Base specialization may need to perform. Scan 
the entire list of tasks now, before completing any portion of this 
questionnaire . While scanning the tasks , think about who can verify the 
frequency with which you have performed the tasks and your level of 

competence for the tasks -- for example, a current or former supervisor. 

When you have scanned the tasks, return to these directions. 



B. On pages 6 and 7 is a form entitled "Verifiers of Frequency and 
Competence". STOP NOW and complete that form. DO NOT complete any 
portion of this questionnaire until you have completed the form 
entitled "Verifiers of Frequency and Competence". When you have 
completed that form, return to these directions. 



C. AFTER you have completed the "Verifiers of Frequency^ and 
Competence" form, place the machine readable answer paper and the "Task 
Listing" before you. For each task in the "Task Listing' , indicate ON 
THE MACHINE READABLE ANSWER PAPER whether or not you have performed that 
task since April 17, 1991 . Use the following scale: 



Have Not Performed /Have P erformed: 

0 = 1 have not performed this task since April 17, 1991 

1 = I have performed this task since April 17, 1991 



If you have NOT performed a particular task since April 17, ”5991 » 
(i.e., you marked "0" on the machine readable answer paper) , goon to the 

next task . 

If you HAVE performed a particular task since April 17, 1991 (i-® • » 
you marked "1" on the machine readable answer paper), indicate ON THE 
MACHINE READABLE ANSWER PAPER the following additional information 
regarding that task: 
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Frequency: How frequently (how many separate times) have you 

performed that task since April 17, 1991? (If you have performed 
that task more than 15 times, mark the 15+ bubble.) 

Competence: What is your level of competence in that task? Select 
the statement below which besw describes your level of ability in 
performing the task: 

1 = Although I have performed this task in the past , I am not 

familiar with how to do it now. 

2 = I have the ability to perform the routine parts of this 

task, but would need guidance in completing other parts 
of the task . 

3 = I have the ability to independently perform all parts of 

this task , and can handle nearly all situations without 
guidance. 

4 = I have an exceptional ability to perform this task. I am 

considered to be a master in this task, and am consulted 
by or direct others in carrying out even the most 
complicated aspects of this task. 

Verifiers: Who (of the people that you listed on the "Verifiers of 
Frequency and Competence" form already) can verify your claimed 
frequency and level of competence for that task? 

For each task, fill in the bubble (i through 5) corresponding to 
the person on your "Verifiers of Frequency and Competence" form 
who can verify your frequency and level of competence for that 
task. For the Verifier(s) question only, you may fill in as many 
bubbles as you have Verifiers for a given task (e.g., if both 
person #2 and person #3 can verify your frequency and level of 
competence for a particular task, you should fill in both Verifier 
bubble #2 and Verifier bubble #3 for that task) . 

If none of the first three Verifiers can verify your frequency and 
level of competence for that task, but SOMEONE ELSE CAN, mark 
Verifier bubble #4 on your machine readable answer paper for that 
task, AND COMPLETE THE FORM ON PAGES 8 TO 10 ENTITLED, 
"IDENTIFICATION OF VERIFIER #4" . 

If NO ONE can verify your frequency and level of competence for 
that task , mark Verifier bubble #5 on your machine readable answer 
paper for that task, AND COMPLETE THE FORM ON PAGE 11 ENTITLED, 
"EXPLANATION FOR LACK OF VERIFIER" . 




2 



You must indicate a person who can verify the frequency and 
level of competence that you claim for each task which you 
claim to have done. Failure to indicate someone who can verify 
your claimed frequency and level of competence (or, in the case of 
Verifier #5, failure to provide an acceptable explanation) may 
result in your not receiving credit for that task. 



D Check to make sure that you have responded on the machine readable 
answer paper to all 20 tasks. For each task, you should have indicated 
whether or not you have performed that task since April 17, 1991 . For 
each task for which the answer was " 1 " (yes) , you should have indicated 
how frequently you have performed that task, your level of competence 
for that task, and one or more persons who can verify your frequency and 
level of competence for that task. 



E . When you have responded to all 20 tasks , continue with the Training 
portion of this questionnaire, beginning on page 12. 



Task Listing 

DATA BASE 



Description 

Prepares a conceptual data base model. 

Prepares entity relationships. 

Defines data base relationships and logical data 
structure diagrams. 

Defines data elements for new data base systems and/or 
normalizes data elements. 

Documents record layouts as they relate to data base 
records. 

Defines technical considerations of data base file 
organization techniques and access methods, including 
storage considerations. 

Reviews proposed applications designs for DBMS 
performance implications. 

Defines data and/or verifies data base descriptions 
(schema) . 

Designs, codes, and implements data base access paths, 

data base interfaces, and specialized data base access 

routines . 

Allocates and initializes physical data base files for 
testing and/or production. 

Analyzes data base performance statistics and fine 
tunes data base . 

Monitors data base mass storage utilizations and 
projects trends. 

Resizes data base areas and/or tables. 




14 Uses DBMS utilities to dump/recover files. 

15 Reorganizes data bases , 

16 Selects and activates DBMS security features. 

17 Maintains and updates data dictionary. 

18 Assists in evaluation and selection of DBMS and CASE 
software products, identifying advantages and 
disadvantages. 

19 Performs and/or coordinates data base software upgrades 
and maintenance . 

20 Works with DBMS vendors to resolve problems regarding 
data base software. 
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Verifiers of Frequency and Competence 

Skim the Task Listing. Who can verify your frequency of 
performance and your level of competence in either some or all of the 
listed tasks? List below UP TO THREE people who can verify your 
frequency and level of competence. List only those people who, by 
virtue of their relationship to you while you learned or performed the 
tasks, are familiar with your experience in the tasks •• for example, 
current or former supervisors. For each Verifier, please PRINT or TYPE 
the information requested below: Illegible entries may result in your 
not receiving credit for some tasks. 

The NYS Department of Civil Service reserves the right to contact 
the people you list for verification of your ratings, and to change your 
ratings if it is determined that you exaggerated either the frequency 
with which you have performed a task or your claimed level of competence 
for that task. 

Verifier #1 

Name: - 

Title: - 

Relationship to you: 

Address: - — — 



Phone: 

Verifier #2 

Name: 

Title: ________ 

Relationship to you: 
Address: 



Phone : 
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Verif ien ^ 

Name: _ 

Title: _________ 

Relationship to you: 



Address: 



Phone : 



ypri f ier #4 = Someone other than the above three people can verify your 
claimed frequency and level of competence for a task. 



One or more of the above three people should be able to verify your 
frequency and level of competence for all or most of the tasks for which 

vou claim experience. If, however, your experience in a particular task 
can be verified only by some OTHER person, indicate 
Verifier bubble #4 on your machine readable answer paper , AND PROVIDING 
THE IDENTIFYING INFORMATION REQUESTED ON THE FORM ON PAGES 8 TO 10 
ENTITLED, "Identification of Verifier #4" . 



Vftrifier #5 = NO ONE can verify your claimed frequency and level of 
competence for a task. 



Someone should be able to verify your frequency and level of 
competence for each task for which you claim experience . If , for some 
reason, there is a task for which NO ONE can verify experience, 
indicate this bv marking Verifier bubble #5 on your machine readable 
^nsw^r paper! aXd ?RWIDING AN ACCEPTABLE EXPLANATION ON THE FORM ON 
PAGE 11 ENTITLED, "Explanation for Lack of Verifier . 
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Identification of Verifier #4 



For each task for which you indicated that someone OTHER than the three 
people listed on your "Verifiers of Frequency and Competence" form can 
verify your claimed frequency and level of competence (i.e. , you marked 
Verifier bubble #4 on your machine readable answer paper) , fill in the 
task number and the requested identifying information below. Please 
PRINT or TYPE your entries : Incomplete or illegible entries may result 
in your not receiving credit for some tasks. 

Task # Information about Verifier #4 (This Verifier may be 
different for different tasks) 



# Name : 

Title: 



Relationship to you: 
Address: 



Phone : 



#_ Name : 

Title : 

Relationship to you: 
Address: 



Phone : 



ERIC 



8 

‘10 



X 



Name: 

Title: ■ 
Relationship to you: 
Address: 



Phone : 



Name: 

Title: 

Relationship to you: 
Address: 



Phone : 



# Name: 

Title: 

Relationship to you: 
Address: 



Phone : 



# Name : 

Title : 

Relationship to you: 
Address: 



Phone: 



If you need more space, attach additional sheets, using the same format 





Explanation for Lack of Verifier 

Incomplete or illegible explanations may result xn your not receiving 
credit for some tasks. 

The NYS Department of Civil Service reserves the right to deny 
credil forany task which lacks a justifiable reason for the lack of a 

Verifier. 

Task # Fxplanation 





If you need more space , attach additional sheets , using the same format 

ft 



Before completing thie Training portion of the questionnaire, you should have coapleted 
the Experience portion of the questionnaire. If you have not ccapleted the Experience 
portion, do so now, before completing this Training portion. 

DIRECTIONS FOR TRAINING PORTION OF THE QUESTIONNAIRE 

Please provide the information requested in the spaces below. When 
indicated, mark the appropriate bubbles on the separate machine 
readable answer paper enclosed with this questionnaire. You will be 
given credit for coursework and training that meets acceptable 
standards determined by the Department of Civil Service. You will not 
receive credit for coursework and training deemed not appropriate for 
the specialty under which you list it. 

Acceptable training in the Data Base technical specialty would have 
covered topics such as: conceptual design of data bases; data 
structures (logical design); facilities planning; data base 
administration; data bases for multi -user networks (not for stand-alone 
pc applications) . Training directed at applications development will 
NOT be accepte J. 

21 . Have you completed any college coursework in the Data Base 
specialty? (College courses are acceptable regardless of the date 
on which they were completed . ) 

Mark the appropriate bubble ("no" or "yes") next to question #21 
on the separate machine readable answer paper. 

If your response is no, proceed to question #23 on this 
questionnaire . 

If your response is yes, indicate below the name(s) of the 
course(s) , a brief description of what the course(s) covered, the 
name and location of the College where you took the course(s) , and 
the number of credits received for each course. If you need more 
space, attach additional sheets, using the same format. 



i) Course Name: 

Course Description: 



Name and Location of College : 



Number of Credits for Course: 





ii) Course Name: 

Course Description: 



Name and Location of College : 



Number of Credits for Course: 



iii) Course Name: 

Course Description: 



Name and Location of College : 



Number of Credits for Course: _ 



iv) Course Name: 

Course Description: 



Name and Location of College : 



Number of Credits for Course: 



22. Add up the number of college credits received for the Data Base 
courses listed above , and write it here : — Credits . 

Record this total by marking the appropriate bubble (’‘1-2" , “3-5" . 

“6-11", or "12 or more") next to question #22 on the machine 
readable answer paper. 



23 .qinne April 17. 1991 . have you completed any non -college credit 
(vendor provided ; ^ployer provided ; college non -credit ; private 
training institution) training in the Data Base specialty? 

To answer "yes" , the training must have been completed s^ce April 
17, 1991 . In addition, the training must have lasted at LEAST one 
full day (at least 6 contact hours) , and may NOT have been a vendor 
product announcement or demonstration . Attendance at conf erences 
with multiple workshops of less than one day is also NO 
acceptable. 



Mark the appropriate bubble ("no" or "yes") next to question #23 
on the machine readable answer paper. 

If your response to question #23 is "no" j you have now completed 
the Training portion of the questionnaire for the Data Base 
specialty. Check to make sure that you have completed the 
Experience portion of the questionnaire. Then follow the 
instructions in the "General Directions" for returning your 
machine readable answer paper and completed questionnaire to the 
NYS Department of Civil Service. 

If your response to question #23 is "yes", provide in the spaces 
below a description of what the training covered, the duration of 
the training (number of contact hours) , the date(s) on which you 
received the training, the name and telephone number of the 
organization sponsoring or providing the training, and the name, 
address, and telephone number of some person (not a relative, co- 
worker, or friend) who.can verify that you received the training. 
List ONLY training that meets the previously described criteria 
of recency, length, and sponsorship. If you need more space, 
attach additional sheets, using the same format. 

Training Description: 



Duration of Training: contact hours 

Date(s) of Training: * 

Name of Training Sponsor/Provider : 

Telephone: 

Name of Verifier: 

Address of Verifier: 

Telephone number of Verifier: 



ii) Training Description: 



Duration of Training: contact hours 

Date(s) of Training: 

Name of Training Sponsor/Provider: 

Telephone: 

Name of Verifier: 

Address of Verifier: 

Telephone number of Verifier: — _ — 

iii) Training Description: — 



Duration of Training: contact hours 

Date(s) of Training: 

Name of Training Sponsor/Provider: 

Telephone: 

Name of Verifier: 

Address of Verifier: — — 

Telephone number of Verifier: 
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iv) Training Description : 



Duration of Training: contact hours 

Date(s) of Training: 

Name of Training Sponsor/Provider: 

Telephone: 

Name of Verifier: 

Address of Verifier: 

Telephone number of Verifier: 

V) Training Description: 



Duration of Training: contact hours 

Date(s) of Training: 

Name of Training Sponsor/Provider: 

Telephone: 

Name of Verifier: 

Address of Verifier: 



Telephone number of Verifier: 



24. Add up the number of contact hours of Data Base training listed 
above, and write it here: contact hours. 

Record this total by marking the appropriate bubble ( "1 -11 " , "12- 
23" , "24-35" , "36-47" , or "48 or more" ) next to question #24 on the 
separate machine readable answer paper. 
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You have now completed the Training portion of the questionnaire for the 
Data Base specialty. Check to make sure that you have also completed the 
Experience portion. Then follow the instructions in the "General 
Directions" for returning your machine readable answer paper and 
completed questionnaire to the NYS Department of Civil Service. 
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BEST COPY AVAILABLE 



